Credit Card Users Churn Prediction

Thera bank recently saw a steep decline in the number of credit card users. Credit cards are a good source of income for banks because they can charge a variety of fees to the users. Some fees are charged to every user irrespective of usage, while others are charged under specified circumstances.

The loss of credit card clients is affecting the bank's bottom line, so the bank wants to analyze the data of customers and identify the clients who will end their credit card services, so the bank can attempt to retain them as credit card clients.

As a Data scientist for Thera bank, I need to derive a classification model that will help the bank improve its services so that customers do not renounce their credit cards.

Objective

• Explore and visualize the dataset.
• Build a classification model to predict if the customer is going to churn or not
• Optimize the model using appropriate techniques
• Generate a set of insights and recommendations that will help the bank

# import the libraries

import numpy as np
import pandas as pd

# Libraries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# libraries to build models
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import GradientBoostingClassifier
from xgboost import XGBClassifier

# To tune model, get different metric scores and split data
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import (
    f1_score,
    accuracy_score,
    recall_score,
    precision_score,
    confusion_matrix,
    roc_auc_score,
    plot_confusion_matrix,
)

# To oversample and undersample data
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler

# To suppress the warnings
import warnings
warnings.filterwarnings("ignore")

churn = pd.read_csv('BankChurners.csv')
df = churn.copy(deep = True)
df.shape

(10127, 21)

df.head(20)

	CLIENTNUM	Attrition_Flag	Customer_Age	Gender	Dependent_count	Education_Level	Marital_Status	Income_Category	Card_Category	Months_on_book	...	Months_Inactive_12_mon	Contacts_Count_12_mon	Credit_Limit	Total_Revolving_Bal	Avg_Open_To_Buy	Total_Amt_Chng_Q4_Q1	Total_Trans_Amt	Total_Trans_Ct	Total_Ct_Chng_Q4_Q1	Avg_Utilization_Ratio
0	768805383	Existing Customer	45	M	3	High School	Married	$60K - $80K	Blue	39	...	1	3	12691.0	777	11914.0	1.335	1144	42	1.625	0.061
1	818770008	Existing Customer	49	F	5	Graduate	Single	Less than $40K	Blue	44	...	1	2	8256.0	864	7392.0	1.541	1291	33	3.714	0.105
2	713982108	Existing Customer	51	M	3	Graduate	Married	$80K - $120K	Blue	36	...	1	0	3418.0	0	3418.0	2.594	1887	20	2.333	0.000
3	769911858	Existing Customer	40	F	4	High School	NaN	Less than $40K	Blue	34	...	4	1	3313.0	2517	796.0	1.405	1171	20	2.333	0.760
4	709106358	Existing Customer	40	M	3	Uneducated	Married	$60K - $80K	Blue	21	...	1	0	4716.0	0	4716.0	2.175	816	28	2.500	0.000
5	713061558	Existing Customer	44	M	2	Graduate	Married	$40K - $60K	Blue	36	...	1	2	4010.0	1247	2763.0	1.376	1088	24	0.846	0.311
6	810347208	Existing Customer	51	M	4	NaN	Married	$120K +	Gold	46	...	1	3	34516.0	2264	32252.0	1.975	1330	31	0.722	0.066
7	818906208	Existing Customer	32	M	0	High School	NaN	$60K - $80K	Silver	27	...	2	2	29081.0	1396	27685.0	2.204	1538	36	0.714	0.048
8	710930508	Existing Customer	37	M	3	Uneducated	Single	$60K - $80K	Blue	36	...	2	0	22352.0	2517	19835.0	3.355	1350	24	1.182	0.113
9	719661558	Existing Customer	48	M	2	Graduate	Single	$80K - $120K	Blue	36	...	3	3	11656.0	1677	9979.0	1.524	1441	32	0.882	0.144
10	708790833	Existing Customer	42	M	5	Uneducated	NaN	$120K +	Blue	31	...	3	2	6748.0	1467	5281.0	0.831	1201	42	0.680	0.217
11	710821833	Existing Customer	65	M	1	NaN	Married	$40K - $60K	Blue	54	...	2	3	9095.0	1587	7508.0	1.433	1314	26	1.364	0.174
12	710599683	Existing Customer	56	M	1	College	Single	$80K - $120K	Blue	36	...	6	0	11751.0	0	11751.0	3.397	1539	17	3.250	0.000
13	816082233	Existing Customer	35	M	3	Graduate	NaN	$60K - $80K	Blue	30	...	1	3	8547.0	1666	6881.0	1.163	1311	33	2.000	0.195
14	712396908	Existing Customer	57	F	2	Graduate	Married	Less than $40K	Blue	48	...	2	2	2436.0	680	1756.0	1.190	1570	29	0.611	0.279
15	714885258	Existing Customer	44	M	4	NaN	NaN	$80K - $120K	Blue	37	...	1	2	4234.0	972	3262.0	1.707	1348	27	1.700	0.230
16	709967358	Existing Customer	48	M	4	Post-Graduate	Single	$80K - $120K	Blue	36	...	2	3	30367.0	2362	28005.0	1.708	1671	27	0.929	0.078
17	753327333	Existing Customer	41	M	3	NaN	Married	$80K - $120K	Blue	34	...	4	1	13535.0	1291	12244.0	0.653	1028	21	1.625	0.095
18	806160108	Existing Customer	61	M	1	High School	Married	$40K - $60K	Blue	56	...	2	3	3193.0	2517	676.0	1.831	1336	30	1.143	0.788
19	709327383	Existing Customer	45	F	2	Graduate	Married	abc	Blue	37	...	1	2	14470.0	1157	13313.0	0.966	1207	21	0.909	0.080

20 rows × 21 columns

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10127 entries, 0 to 10126
Data columns (total 21 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   CLIENTNUM                 10127 non-null  int64  
 1   Attrition_Flag            10127 non-null  object 
 2   Customer_Age              10127 non-null  int64  
 3   Gender                    10127 non-null  object 
 4   Dependent_count           10127 non-null  int64  
 5   Education_Level           8608 non-null   object 
 6   Marital_Status            9378 non-null   object 
 7   Income_Category           10127 non-null  object 
 8   Card_Category             10127 non-null  object 
 9   Months_on_book            10127 non-null  int64  
 10  Total_Relationship_Count  10127 non-null  int64  
 11  Months_Inactive_12_mon    10127 non-null  int64  
 12  Contacts_Count_12_mon     10127 non-null  int64  
 13  Credit_Limit              10127 non-null  float64
 14  Total_Revolving_Bal       10127 non-null  int64  
 15  Avg_Open_To_Buy           10127 non-null  float64
 16  Total_Amt_Chng_Q4_Q1      10127 non-null  float64
 17  Total_Trans_Amt           10127 non-null  int64  
 18  Total_Trans_Ct            10127 non-null  int64  
 19  Total_Ct_Chng_Q4_Q1       10127 non-null  float64
 20  Avg_Utilization_Ratio     10127 non-null  float64
dtypes: float64(5), int64(10), object(6)
memory usage: 1.6+ MB

# Changing objects to categorical, getting value counts for only categorical variables
cat_cols = ['Attrition_Flag','Gender','Education_Level','Marital_Status','Income_Category','Card_Category']
for col in cat_cols:
    if df[col].dtype == 'object':
        df[col] = pd.Categorical(df[col])

for col in cat_cols:
    vc_col = df[col].value_counts()
    print(vc_col)

Existing Customer    8500
Attrited Customer    1627
Name: Attrition_Flag, dtype: int64
F    5358
M    4769
Name: Gender, dtype: int64
Graduate         3128
High School      2013
Uneducated       1487
College          1013
Post-Graduate     516
Doctorate         451
Name: Education_Level, dtype: int64
Married     4687
Single      3943
Divorced     748
Name: Marital_Status, dtype: int64
Less than $40K    3561
$40K - $60K       1790
$80K - $120K      1535
$60K - $80K       1402
abc               1112
$120K +            727
Name: Income_Category, dtype: int64
Blue        9436
Silver       555
Gold         116
Platinum      20
Name: Card_Category, dtype: int64

Data Dictionary

• CLIENTNUM: Client number.
  • Different for each client, will likely be dropped
• Attrition_Flag: Did client drop credit card services? ("Attrited Customer" or "Existing Customer")
  • Target variable
  • Appears imbalanced - 1667 Attrited, 8500 Existing. May need to perform up- or down-sampling
• Customer_Age: Age in Years
• Gender: Gender of account holder ("F" or "M")
• Dependent_count: Number of dependents
• Education_Level: Highest level of education achieved. ("Graduate", "High School", "Uneducated", "College" (student), "Post-Graduate", or "Doctorate")
  • MISSING from 1519 rows
  • Categorical variable, will likely replace missing with mode, "Graduate", with 3128 instances
• Marital_Status: Marital Status ("Married", "Single", or "Divorced")
  • MISSING from 749 rows
  • Categorical variable, will likely replace missing with mode, "Married", with 4687 instances
• Income_Category: Annual Income Category of account holder ("Less than 40K", "40K - 60K", "60K - 80K", "80K - 120K", "120K +" or "abc")
  • abc can be considered as missing values
  • Categorical variable, will likely replace missing with mode, Less than 40K, with 3561 instances
• Card_Category: Type of Card ("Blue", "Silver", "Gold", or "Platinum")
• Months_on_book: How long has the client been with us?
• Total_Relationship_Count: How many bank products does the client have?
• Months_Inactive_12_mon: How many months has the client been inactive in the last 12 months?
• Contacts_Count_12_mon: How many contacts exist between client and bank in the last 12 months?
• Credit_Limit: Credit Limit on the Card
• Total_Revolving_Bal: Balance that carries over from month to month
• Avg_Open_To_Buy: Amount of credit remaining (Average of last 12 months)
• Total_Trans_Amt: Total Transaction Amount (Last 12 months)
• Total_Trans_Ct: Total Number of Transactions (Last 12 months)
• Total_Ct_Chng_Q4_Q1: Ratio of Total Transaction Counts, 4th Quarter to 1st Quarter
• Total_Amt_Chng_Q4_Q1: Ratio of Total Transaction Amount, 4th Quarter to 1st Quarter
• Avg_Utilization_Ratio: Represents how much of the available credit the client spent

# Statistical analysis
df.describe()

	CLIENTNUM	Customer_Age	Dependent_count	Months_on_book	Total_Relationship_Count	Months_Inactive_12_mon	Contacts_Count_12_mon	Credit_Limit	Total_Revolving_Bal	Avg_Open_To_Buy	Total_Amt_Chng_Q4_Q1	Total_Trans_Amt	Total_Trans_Ct	Total_Ct_Chng_Q4_Q1	Avg_Utilization_Ratio
count	1.012700e+04	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000	10127.000000
mean	7.391776e+08	46.325960	2.346203	35.928409	3.812580	2.341167	2.455317	8631.953698	1162.814061	7469.139637	0.759941	4404.086304	64.858695	0.712222	0.274894
std	3.690378e+07	8.016814	1.298908	7.986416	1.554408	1.010622	1.106225	9088.776650	814.987335	9090.685324	0.219207	3397.129254	23.472570	0.238086	0.275691
min	7.080821e+08	26.000000	0.000000	13.000000	1.000000	0.000000	0.000000	1438.300000	0.000000	3.000000	0.000000	510.000000	10.000000	0.000000	0.000000
25%	7.130368e+08	41.000000	1.000000	31.000000	3.000000	2.000000	2.000000	2555.000000	359.000000	1324.500000	0.631000	2155.500000	45.000000	0.582000	0.023000
50%	7.179264e+08	46.000000	2.000000	36.000000	4.000000	2.000000	2.000000	4549.000000	1276.000000	3474.000000	0.736000	3899.000000	67.000000	0.702000	0.176000
75%	7.731435e+08	52.000000	3.000000	40.000000	5.000000	3.000000	3.000000	11067.500000	1784.000000	9859.000000	0.859000	4741.000000	81.000000	0.818000	0.503000
max	8.283431e+08	73.000000	5.000000	56.000000	6.000000	6.000000	6.000000	34516.000000	2517.000000	34516.000000	3.397000	18484.000000	139.000000	3.714000	0.999000

Observations

• As previously noted, "CLIENTNUM" is not statistically significant and will be dropped.
• "Age" column is not significantly skewed.
  • Both mean and median are around 46.
  • Range is from 26-73, so outliers are unlikely.
• "Dependent_count" is slightly right-skewed.
  • Mean (2.346) is greater than median (2), and, although they are close, the small range means that even a small difference counts.
  • Range is 1-5, so outliers are unlikely.
• "Months_on_book" is not significantly skewed.
  • Both mean and median are around 36.
  • Range is 13-56 - outliers unlikely.
• "Total_Relationship_Count" is slightly left-skewed.
  • Mean (3.813) is less than median (4), and, although they are close, the small range means that even a small difference counts.
  • Range is 1-6, so outliers are unlikely.
• "Months_Inactive_12_mon" is slightly right-skewed.
  • Small range, so the small difference between mean (2.341) and median (2) is still significant.
  • Range is 0-6, so outliers are unlikely.
• "Contacts_Count_12_mon" is slightly right-skewed.
  • Small range, so the small difference between mean (2.455) and median (2) is still significant.
  • Range is 0-6, outliers are unlikely.
  • Possibly directly correlated with "Months_Inactive_12_mon".
• "Credit_Limit" is significantly right-skewed.
  • Mean (8631.954) is higher than median (4549).
  • Range is 1438-34516, so outliers are likely, particularly on the high end.
• "Total_Revolving_Bal" is slightly left-skewed
  • Mean (1162.814) is less than median (1276).
  • Range is 0-2517, outliers are unlikely.
• "Avg_Open_To_Buy" is significantly right-skewed.
  • Mean (9090.685) is higher than median (3474).
  • Range is 3-34516, so outliers are likely, particularly on the high end.
  • Possibly directly correlated with "Credit_Limit"
• "Total_Amt_Chng_Q4_Q1" is slightly right-skewed.
  • Mean (0.760) is slightly greater than median (0.736).
  • Range is 0-3.397, so outliers are likely on both sides.
• "Total_Trans_Amt" is right skewed.
  • Median (4404.086) is greater than median (3899)
  • Range is 510-18484, so outliers are likely on the high end.
• "Total_Trans_Ct" is slightly left-skewed.
  • Mean (64.859) is slightly less than median (67).
  • Range is 10-139, so outliers are possible on the high end.
• "Total_Ct_Chng_Q4_Q1" is not significantly skewed.
  • Mean and median are both around 71.
  • Range is 0-3.714, so outliers are likely on both sides.
  • Possibly directly correlated with "Total_Amt_Chng_Q4_Q1"
• "Avg_Utilization_Ratio" is right skewed.
  • Mean (0.276) is greater than median (0.176)
  • Range is 0-0.999, so outliers may exist.
• Several columns may have outliers, so I will run box plots to check, and treat outliers, if necessary.
• Several columns may be correlated, so I will run a heat map to check

# dropping "CLIENTNUM", confirming that it's gone and that objects are now
# encoded as categorical
df.drop(["CLIENTNUM"], axis=1, inplace=True)
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10127 entries, 0 to 10126
Data columns (total 20 columns):
 #   Column                    Non-Null Count  Dtype   
---  ------                    --------------  -----   
 0   Attrition_Flag            10127 non-null  category
 1   Customer_Age              10127 non-null  int64   
 2   Gender                    10127 non-null  category
 3   Dependent_count           10127 non-null  int64   
 4   Education_Level           8608 non-null   category
 5   Marital_Status            9378 non-null   category
 6   Income_Category           10127 non-null  category
 7   Card_Category             10127 non-null  category
 8   Months_on_book            10127 non-null  int64   
 9   Total_Relationship_Count  10127 non-null  int64   
 10  Months_Inactive_12_mon    10127 non-null  int64   
 11  Contacts_Count_12_mon     10127 non-null  int64   
 12  Credit_Limit              10127 non-null  float64 
 13  Total_Revolving_Bal       10127 non-null  int64   
 14  Avg_Open_To_Buy           10127 non-null  float64 
 15  Total_Amt_Chng_Q4_Q1      10127 non-null  float64 
 16  Total_Trans_Amt           10127 non-null  int64   
 17  Total_Trans_Ct            10127 non-null  int64   
 18  Total_Ct_Chng_Q4_Q1       10127 non-null  float64 
 19  Avg_Utilization_Ratio     10127 non-null  float64 
dtypes: category(6), float64(5), int64(9)
memory usage: 1.1 MB

# Statistical summary of the categorical columns
df.describe(exclude=np.number).T

	count	unique	top	freq
Attrition_Flag	10127	2	Existing Customer	8500
Gender	10127	2	F	5358
Education_Level	8608	6	Graduate	3128
Marital_Status	9378	3	Married	4687
Income_Category	10127	6	Less than $40K	3561
Card_Category	10127	4	Blue	9436

Observations

• Modes, as previously noted, are "Existing Customer" for "Attrition_Flag"; "F" for "Gender; "Married" for "Marital_Status"; "Less than 40K" for "Income_Category"; and "Blue" for "Card_Category"
• Missing data was also previously noted. Missing values will be replaced with the mode, but not until we split data into train/test/validation, in order to prevent data leakage

EDA

Uni-variate

• I will create histogram/box-plot combos for numeric data
• I will create bar plots for categorical data, and any numeric data that might look better that way
• A note about my philosophy on outliers:
  • In general, I feel that outliers should be left in if they are justifiably real data
  • The options are to either take out the rows, which would result in losing good data from other variables, or treat the outliers, which removes data that is real and replaces it with data that is fake.
  • In the past, I have lost marks for not treating outliers, regardless of my justifications for not doing so. I hope that you take this into consideration as you grade this project.

# Function to create a combo boxplot/histogram with same x-axis.

def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to the show density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,
        sharex=True,  # x-axis will be shared
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # star on mean value of the column for the boxplot
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Green line on mean for the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Black line on median for the histogram

histogram_boxplot(df, "Customer_Age")

• As previously noted, not skewed
• Normal distribution
• Two high outliers. THESE SEEM JUSTIFIED - it is not surprising to see customers in their 70s. Therefore, since the data is justifiably real, and not a mistake, I WILL NOT TREAT THESE OUTLIERS.

histogram_boxplot(df, "Dependent_count")

• Slightly right-skewed with no outliers, as previously noted
• This variable would look better as a bar plot.

histogram_boxplot(df, "Months_on_book")

• Mostly normal distribution, although there is a very high mode around 36
• Five low and three high outliers. THESE SEEM JUSTIFIED - it is normal to expect that we would have some very old and very new customers. Therefore, since the data is justifiably real, and not a mistake, I WILL NOT TREAT THESE OUTLIERS.

histogram_boxplot(df, "Total_Relationship_Count")

• Slightly left-skewed with no outliers, as previously noted
• This variable would look better as a bar plot.

histogram_boxplot(df, "Months_Inactive_12_mon")

• Slightly right-skewed, as previously noted
• I previously predicted no outliers; however, my box plot shows outliers at 0, 5 and 6.
• THESE OUTLIERS SEEM JUSTIFIED - I would expect to see some customers who were not inactive at all, and other customers who were inactive for 5 or 6 months. Therefore, since the data is justifiably real, and not a mistake, I WILL NOT TREAT THESE OUTLIERS
• This variable would look better as a bar plot.

histogram_boxplot(df, "Contacts_Count_12_mon")

• Slightly right-skewed, as previously noted
• I previously predicted no outliers; however, my box plot shows outliers at 0, 5 and 6.
• THESE OUTLIERS SEEM JUSTIFIED - I would expect to see some customers who were not contacted at all, and other customers who were contacted 5 or 6 times. Therefore, since the data is justifiably real, and not a mistake, I WILL NOT TREAT THESE OUTLIERS
• This variable would look better as a bar plot.

histogram_boxplot(df, "Credit_Limit")

# Distribution is not normal.  I will try taking the log of this variable to see if it normalizes

df['Credit_Limit_Log'] = np.log(df['Credit_Limit'])
histogram_boxplot(df, 'Credit_Limit_Log')

• Distribution is now closer to normal
• Right skewed, as previously noted
• Three modes, one where it should be, and two on the minimum and maximum.
• It makes sense that there would be a lot of people with the lowest and highest credit limits. There is probably a ceiling and a floor that customers are not allowed to go below or above. Customers that should have had a higher or lower limit get capped at this floor and ceiling.
• Before the log transformation, there were many high outliers; these have been fixed with the transformation, and they are justifiably real numbers, so THESE OUTLIERS WILL NOT BE TREATED

histogram_boxplot(df, "Total_Revolving_Bal")

• Left skewed, as previously noted
• No outliers
• Two modes, one at the minimum, and one at the maximum. These make sense, as there is probably a floor and ceiling, much like the "Credit_Limit" variable.
• It looks to be normally distributed, and would look even closer to normal without the modes at the extremes.

histogram_boxplot(df, "Avg_Open_To_Buy")

# Not normal, will try a log transformation
df['Avg_Open_To_Buy_Log'] = np.log(df['Avg_Open_To_Buy'])
histogram_boxplot(df, 'Avg_Open_To_Buy_Log')

# That doesn't look much better.  Trying with the square root
df['Avg_Open_To_Buy_Sqrt'] = np.sqrt(df['Avg_Open_To_Buy'])
histogram_boxplot(df, 'Avg_Open_To_Buy_Sqrt')

• Much more closer to normal, right-skewed as previously noted.
• Outliers on the high side have been fixed with the sqrt transformation.
• High mode is justifiable - it is to be expected that a higher number of people would have a higher average amount of credit left, if they aren't using their credit card much.
• For the same reason, I WILL NOT TREAT OUTLIERS, as it is justifiably real data, and they are no longer outliers once I take the square root

histogram_boxplot(df, "Total_Amt_Chng_Q4_Q1")

• Normal distribution, slightly right skewed, as previously noted.
• Several outliers on the low side, many on the high side
• THESE OUTLIERS SEEM JUSTIFIED - I would expect to see some customers with more transactions in the fourth quarter than the first, as well as some with more in the first quarter than the fourth. Therefore, since the data is justifiably real, and not a mistake, I WILL NOT TREAT THESE OUTLIERS

histogram_boxplot(df, "Total_Trans_Amt")

• Right skewed, as previously noted
• Four modes, not normally distributed
• Outliers are justifiable, as it would make sense that customers with a higher credit limit would have a higher transaction amount. Therefore, since the data is justifiably valid, I WILL NOT TREAT THESE OUTLIERS

histogram_boxplot(df, "Total_Trans_Ct")

• As previously noted, slightly left skewed
• Outliers on the high end. These are justified - it makes perfect sense that some customers would have that many transactions. Therefore, since the outliers are justifiably real data, I WILL NOT TREAT THESE OUTLIERS

histogram_boxplot(df, "Total_Ct_Chng_Q4_Q1")

• Normal distribution, not skewed, as previously noted
• Outliers on both sides, as previously noted
• THESE OUTLIERS SEEM JUSTIFIED - I would expect to see some customers with more transactions in the fourth quarter than the first, as well as some with more in the first quarter than the fourth. Therefore, since the data is justifiably real, and not a mistake, I WILL NOT TREAT THESE OUTLIERS

histogram_boxplot(df, "Avg_Utilization_Ratio")

# not normally distributed.  I will try taking the log of the variable
# Since there are 0's, and you can't take the log of non-positive numbers, I will add 0.0001 to the
# variable before taking the log:
df['Avg_Utilization_Ratio_Log'] = np.log(df['Avg_Utilization_Ratio'] + 0.01)
histogram_boxplot(df, 'Avg_Utilization_Ratio_Log')

# Still looks strange.  I will try taking the square root:

df['Avg_Utilization_Ratio_Sqrt'] = np.sqrt(df['Avg_Utilization_Ratio'])
histogram_boxplot(df, 'Avg_Utilization_Ratio_Sqrt')

• Original was right-skewed, but square root is not skewed.
• No outliers, which is not what I predicted

# function to create labeled barplots

def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot

labeled_barplot(df, "Dependent_count")

• Most families have 2 or 3 dependents
• The least is 5, and 0 is close, with 1 and 4 in the middle

labeled_barplot(df, "Total_Relationship_Count")

• Most have 3 relationships
• Essentially the same amount of people have 4, 5, or 6 banking relationships
• Comparatively, very few have only 1 relationship with the bank

labeled_barplot(df, "Months_Inactive_12_mon")

• Most people were inactive for 3 months, followed by 2 and 1
• Very few people were inactive for 4-6 months, and almost nobody (29) was inactive for 0 months

labeled_barplot(df, "Contacts_Count_12_mon")

• Most people were had 2-3 contacts with the bank
• 1 and 4 are essentially tied in the middle
• This is followed by 0, 5 and 6. It is more likely to have no contact with the bank than it is to have 5-6 contacts.

labeled_barplot(df, "Attrition_Flag")

• There are approximately 5 times as many existing customers as there are attrited customers
• Since this is the target variable, I am concerned about bias. Therefore, I will explore up- and down-sampling techniques

labeled_barplot(df, "Gender")

• Slightly more men than women

labeled_barplot(df, "Education_Level")

• Most customers graduated college
• Few customers have a doctorate or a post-graduate degree
• In the middle, respectively, are high school graduates, uneducated persons, and college students
• Notably, missing data, not featured, would come in 3rd place with around 1900.

labeled_barplot(df, "Marital_Status")

• Most customers are married, followed closely by singles
• Relatively few customers are divorced

labeled_barplot(df, "Income_Category")

• Most customers make less than 40,000
• The rest are basically even, with the lowest number in the 120,000+ category

labeled_barplot(df, "Card_Category")

• Most customers by far have a Blue card
• Followed by silver, gold, and platinum, which only has 20

Multi-variate

• Strategy is to run a pairplot to see relationships between all numeric variables
• I will also run a heat map to check for directly-correlated columns
• I will run stacked bar plots to compare each categorical variable with the target

# dropping variables that had a log or sqrt transformation, as well as variables created by me
# with a log transformation that was not successful
# Also, converting "Attrition Flag" to numeric, with "Existing Customer" 
# encoded as "0", and "Attrited Customer" encoded as "1"
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10127 entries, 0 to 10126
Data columns (total 25 columns):
 #   Column                      Non-Null Count  Dtype   
---  ------                      --------------  -----   
 0   Attrition_Flag              10127 non-null  category
 1   Customer_Age                10127 non-null  int64   
 2   Gender                      10127 non-null  category
 3   Dependent_count             10127 non-null  int64   
 4   Education_Level             8608 non-null   category
 5   Marital_Status              9378 non-null   category
 6   Income_Category             10127 non-null  category
 7   Card_Category               10127 non-null  category
 8   Months_on_book              10127 non-null  int64   
 9   Total_Relationship_Count    10127 non-null  int64   
 10  Months_Inactive_12_mon      10127 non-null  int64   
 11  Contacts_Count_12_mon       10127 non-null  int64   
 12  Credit_Limit                10127 non-null  float64 
 13  Total_Revolving_Bal         10127 non-null  int64   
 14  Avg_Open_To_Buy             10127 non-null  float64 
 15  Total_Amt_Chng_Q4_Q1        10127 non-null  float64 
 16  Total_Trans_Amt             10127 non-null  int64   
 17  Total_Trans_Ct              10127 non-null  int64   
 18  Total_Ct_Chng_Q4_Q1         10127 non-null  float64 
 19  Avg_Utilization_Ratio       10127 non-null  float64 
 20  Credit_Limit_Log            10127 non-null  float64 
 21  Avg_Open_To_Buy_Log         10127 non-null  float64 
 22  Avg_Open_To_Buy_Sqrt        10127 non-null  float64 
 23  Avg_Utilization_Ratio_Log   10127 non-null  float64 
 24  Avg_Utilization_Ratio_Sqrt  10127 non-null  float64 
dtypes: category(6), float64(10), int64(9)
memory usage: 1.5 MB

df.drop(["Avg_Utilization_Ratio", "Avg_Utilization_Ratio_Log", "Avg_Open_To_Buy",
         "Avg_Open_To_Buy_Log", "Credit_Limit", ], axis=1, inplace=True)
df["Attrition_Flag"].replace("Existing Customer", 0, inplace=True)
df["Attrition_Flag"].replace("Attrited Customer", 1, inplace=True)
df['Attrition_Flag'] = df.Attrition_Flag.astype('int64')
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10127 entries, 0 to 10126
Data columns (total 20 columns):
 #   Column                      Non-Null Count  Dtype   
---  ------                      --------------  -----   
 0   Attrition_Flag              10127 non-null  int64   
 1   Customer_Age                10127 non-null  int64   
 2   Gender                      10127 non-null  category
 3   Dependent_count             10127 non-null  int64   
 4   Education_Level             8608 non-null   category
 5   Marital_Status              9378 non-null   category
 6   Income_Category             10127 non-null  category
 7   Card_Category               10127 non-null  category
 8   Months_on_book              10127 non-null  int64   
 9   Total_Relationship_Count    10127 non-null  int64   
 10  Months_Inactive_12_mon      10127 non-null  int64   
 11  Contacts_Count_12_mon       10127 non-null  int64   
 12  Total_Revolving_Bal         10127 non-null  int64   
 13  Total_Amt_Chng_Q4_Q1        10127 non-null  float64 
 14  Total_Trans_Amt             10127 non-null  int64   
 15  Total_Trans_Ct              10127 non-null  int64   
 16  Total_Ct_Chng_Q4_Q1         10127 non-null  float64 
 17  Credit_Limit_Log            10127 non-null  float64 
 18  Avg_Open_To_Buy_Sqrt        10127 non-null  float64 
 19  Avg_Utilization_Ratio_Sqrt  10127 non-null  float64 
dtypes: category(5), float64(5), int64(10)
memory usage: 1.2 MB

sns.pairplot(data=df, corner = True, diag_kind="kde")

<seaborn.axisgrid.PairGrid at 0x22512b46640>

• High positive correlation between
  • "Months_on_book" and "Customer_Age"
  • "Credit_Limit_Log" and "Avg_Open_To_Buy_Sqrt" (as predicted)
• High negative correlation between:
  • "Credit_Limit_Log" and "Avg_Utilization_Ratio_Sqrt"
  • "Avg_Open_To_Buy_Sqrt" and "Avg_Utilization_Ratio_Sqrt"
• Weak positive correlation between:
  • "Total_Ct_Chng_Q4_Q1" and "Months_on_book"
  • "Total_Ct_Chng_Q4_Q1" and "Customer_Age"
  • "Total_Trans_Amt" and "Total_Trans_Ct"
  • "Total_Ct_Chng_Q4_Q1" and "Total_Amt_Chng_Q4_Q1" (as predicted)
• Uneven positive correlation between "Total_Revolving_Bal" and "Avg_Utilization_Ratio_Sqrt"
• Other patterns exist that are difficult to put into terms of "positive" or "negative" correlation. I will run a heatmap to see if anything emerges.

plt.figure(figsize=(15, 7))
sns.heatmap(df.corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()

• Observations from pairplot are confirmed
• Weak negative correlation between:
  • "Total_Relationship_Count" and "Total_Trans_Ct"
  • "Total_Relationship_Count" and "Total_Trans_Amt"
  • "Attrition_Flag" and "Avg_Utilization_Ratio_Sqrt"
  • "Attrition_Flag" and "Total_Ct_Chng_Q4_Q1"
  • "Attrition_Flag" and "Total_Trans_Ct"
  • "Attrition_Flag" and "Total_Revolving_Bal"
• Weak positive correlation between "Attrition_Flag" and "Contacts_Ct_12_mon"
• "Credit_Limit_Log" and "Avg_Open_To_Buy_Sqrt", at 0.96, are almost perfectly correlated
• I predicted correlation between "Contacts_Count_12_mon" and "Months_Inactive_12_mon", but these variables show no significant correlation
• Other very weak correlations exist, but I did not note any with a correlation less than 0.20

# function to plot stacked bar chart

def stacked_barplot(data, predictor, target):
    """
    Print the category counts and plot a stacked bar chart

    data: dataframe
    predictor: independent variable
    target: target variable
    """
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
    plt.legend(
        loc="lower left",
        frameon=False,
    )
    plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
    plt.show()

stacked_barplot(df, "Gender", "Attrition_Flag")

Attrition_Flag     0     1    All
Gender                           
All             8500  1627  10127
F               4428   930   5358
M               4072   697   4769
------------------------------------------------------------------------------------------------------------------------

• Slightly more females are attrited than males, although the difference is negligible

stacked_barplot(df, "Education_Level", "Attrition_Flag")

Attrition_Flag      0     1   All
Education_Level                  
All              7237  1371  8608
Graduate         2641   487  3128
High School      1707   306  2013
Uneducated       1250   237  1487
College           859   154  1013
Doctorate         356    95   451
Post-Graduate     424    92   516
------------------------------------------------------------------------------------------------------------------------

• Those with a doctorate or post-graduate degree are more likely to be attrited
• The rest are basically even with one another

stacked_barplot(df, "Marital_Status", "Attrition_Flag")

Attrition_Flag     0     1   All
Marital_Status                  
All             7880  1498  9378
Married         3978   709  4687
Single          3275   668  3943
Divorced         627   121   748
------------------------------------------------------------------------------------------------------------------------

• No significant difference between attrited rates based on marital status

stacked_barplot(df, "Income_Category", "Attrition_Flag")

Attrition_Flag      0     1    All
Income_Category                   
All              8500  1627  10127
Less than $40K   2949   612   3561
$40K - $60K      1519   271   1790
$80K - $120K     1293   242   1535
$60K - $80K      1213   189   1402
abc               925   187   1112
$120K +           601   126    727
------------------------------------------------------------------------------------------------------------------------

• High- and low-earners appear to be slightly more likely to attrite, but the difference is negligible

stacked_barplot(df, "Card_Category", "Attrition_Flag")

Attrition_Flag     0     1    All
Card_Category                    
All             8500  1627  10127
Blue            7917  1519   9436
Silver           473    82    555
Gold              95    21    116
Platinum          15     5     20
------------------------------------------------------------------------------------------------------------------------

• Platinum members are most likely to attrite, by a lot
• The other three are basically even, in the order Gold, Blue, Silver

Building the Models

• First, I will split the data, then I will fill missing values on each set
• Next, I will perform one-hot encoding on each set
• Then, I will build the following models:
  • Logistic Regression
  • Decision Tree
  • Bagging Classifier (base estimator will be whichever performs better between Logistic Regression and Decision Tree)
  • Random Forest
  • Gradient Boosting
  • Extreme Gradient Boosting
• I will perform oversampling with SMOTE on each of the above models
• I will perform undersampling with Random Under Sampler on each model
• I will tune the 3 best performers with randomized search
  • Which metric should I use?
  • A false negative in this case means that I predicted the customer would NOT drop their credit card, when in reality they would have. The lost revenue here is a major cost.
  • A false positive in this case means I predicted they would drop their credit card, and they did not, so very low cost.
  • Since false negatives have a much higher cost, Recall, which gives the ratio of true positives to actual positives, is the most important metric. I will calculate accuracy, precision, and F1 as well, but my goal will be to maximize Recall.
• Finally, I will use a pipeline to productionize the best model

X = df.drop(["Attrition_Flag"], axis=1)
y = df["Attrition_Flag"]

# Splitting data into Train, Validation and Test sets
# First, I will split into temporary and test, with 20% in test

X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1, stratify=y
)

# Then, I will split temp set into train and validation

X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=1, stratify=y_temp
)

# finding the size of each dataset
print(X_train.shape, X_val.shape, X_test.shape)

(6075, 19) (2026, 19) (2026, 19)

# To prevent data leakage, I will replace missing values in each set separately

X_train['Education_Level'].fillna("Graduate", inplace = True)
X_val['Education_Level'].fillna("Graduate", inplace = True)
X_test['Education_Level'].fillna("Graduate", inplace = True)
X_train['Marital_Status'].fillna("Married", inplace = True)
X_val['Marital_Status'].fillna("Married", inplace = True)
X_test['Marital_Status'].fillna("Married", inplace = True)
X_train['Income_Category'].replace("abc", "Less than 40K", inplace = True)
X_val['Income_Category'].replace("abc", "Less than 40K", inplace = True)
X_test['Income_Category'].replace("abc", "Less than 40K", inplace = True)

# confirming that all missing data is filled
X_train.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 6075 entries, 800 to 4035
Data columns (total 19 columns):
 #   Column                      Non-Null Count  Dtype   
---  ------                      --------------  -----   
 0   Customer_Age                6075 non-null   int64   
 1   Gender                      6075 non-null   category
 2   Dependent_count             6075 non-null   int64   
 3   Education_Level             6075 non-null   category
 4   Marital_Status              6075 non-null   category
 5   Income_Category             6075 non-null   category
 6   Card_Category               6075 non-null   category
 7   Months_on_book              6075 non-null   int64   
 8   Total_Relationship_Count    6075 non-null   int64   
 9   Months_Inactive_12_mon      6075 non-null   int64   
 10  Contacts_Count_12_mon       6075 non-null   int64   
 11  Total_Revolving_Bal         6075 non-null   int64   
 12  Total_Amt_Chng_Q4_Q1        6075 non-null   float64 
 13  Total_Trans_Amt             6075 non-null   int64   
 14  Total_Trans_Ct              6075 non-null   int64   
 15  Total_Ct_Chng_Q4_Q1         6075 non-null   float64 
 16  Credit_Limit_Log            6075 non-null   float64 
 17  Avg_Open_To_Buy_Sqrt        6075 non-null   float64 
 18  Avg_Utilization_Ratio_Sqrt  6075 non-null   float64 
dtypes: category(5), float64(5), int64(9)
memory usage: 742.4 KB

X_test.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2026 entries, 9760 to 413
Data columns (total 19 columns):
 #   Column                      Non-Null Count  Dtype   
---  ------                      --------------  -----   
 0   Customer_Age                2026 non-null   int64   
 1   Gender                      2026 non-null   category
 2   Dependent_count             2026 non-null   int64   
 3   Education_Level             2026 non-null   category
 4   Marital_Status              2026 non-null   category
 5   Income_Category             2026 non-null   category
 6   Card_Category               2026 non-null   category
 7   Months_on_book              2026 non-null   int64   
 8   Total_Relationship_Count    2026 non-null   int64   
 9   Months_Inactive_12_mon      2026 non-null   int64   
 10  Contacts_Count_12_mon       2026 non-null   int64   
 11  Total_Revolving_Bal         2026 non-null   int64   
 12  Total_Amt_Chng_Q4_Q1        2026 non-null   float64 
 13  Total_Trans_Amt             2026 non-null   int64   
 14  Total_Trans_Ct              2026 non-null   int64   
 15  Total_Ct_Chng_Q4_Q1         2026 non-null   float64 
 16  Credit_Limit_Log            2026 non-null   float64 
 17  Avg_Open_To_Buy_Sqrt        2026 non-null   float64 
 18  Avg_Utilization_Ratio_Sqrt  2026 non-null   float64 
dtypes: category(5), float64(5), int64(9)
memory usage: 248.1 KB

X_val.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2026 entries, 2894 to 6319
Data columns (total 19 columns):
 #   Column                      Non-Null Count  Dtype   
---  ------                      --------------  -----   
 0   Customer_Age                2026 non-null   int64   
 1   Gender                      2026 non-null   category
 2   Dependent_count             2026 non-null   int64   
 3   Education_Level             2026 non-null   category
 4   Marital_Status              2026 non-null   category
 5   Income_Category             2026 non-null   category
 6   Card_Category               2026 non-null   category
 7   Months_on_book              2026 non-null   int64   
 8   Total_Relationship_Count    2026 non-null   int64   
 9   Months_Inactive_12_mon      2026 non-null   int64   
 10  Contacts_Count_12_mon       2026 non-null   int64   
 11  Total_Revolving_Bal         2026 non-null   int64   
 12  Total_Amt_Chng_Q4_Q1        2026 non-null   float64 
 13  Total_Trans_Amt             2026 non-null   int64   
 14  Total_Trans_Ct              2026 non-null   int64   
 15  Total_Ct_Chng_Q4_Q1         2026 non-null   float64 
 16  Credit_Limit_Log            2026 non-null   float64 
 17  Avg_Open_To_Buy_Sqrt        2026 non-null   float64 
 18  Avg_Utilization_Ratio_Sqrt  2026 non-null   float64 
dtypes: category(5), float64(5), int64(9)
memory usage: 248.1 KB

# confirming that "abc" is no longer a value in "Income_Category"
X_train.Income_Category.value_counts()

Less than $40K    2129
$40K - $60K       1059
$80K - $120K       953
$60K - $80K        831
Less than 40K      654
$120K +            449
Name: Income_Category, dtype: int64

X_test.Income_Category.value_counts()

Less than $40K    696
$40K - $60K       370
$60K - $80K       292
$80K - $120K      289
Less than 40K     237
$120K +           142
Name: Income_Category, dtype: int64

X_val.Income_Category.value_counts()

Less than $40K    736
$40K - $60K       361
$80K - $120K      293
$60K - $80K       279
Less than 40K     221
$120K +           136
Name: Income_Category, dtype: int64

# Performing one-hot encoding on categorical variables
X_train = pd.get_dummies(X_train, drop_first = True)
X_test = pd.get_dummies(X_test, drop_first = True)
X_val = pd.get_dummies(X_val, drop_first = True)

# Confirming that it happened
X_train.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 6075 entries, 800 to 4035
Data columns (total 30 columns):
 #   Column                          Non-Null Count  Dtype  
---  ------                          --------------  -----  
 0   Customer_Age                    6075 non-null   int64  
 1   Dependent_count                 6075 non-null   int64  
 2   Months_on_book                  6075 non-null   int64  
 3   Total_Relationship_Count        6075 non-null   int64  
 4   Months_Inactive_12_mon          6075 non-null   int64  
 5   Contacts_Count_12_mon           6075 non-null   int64  
 6   Total_Revolving_Bal             6075 non-null   int64  
 7   Total_Amt_Chng_Q4_Q1            6075 non-null   float64
 8   Total_Trans_Amt                 6075 non-null   int64  
 9   Total_Trans_Ct                  6075 non-null   int64  
 10  Total_Ct_Chng_Q4_Q1             6075 non-null   float64
 11  Credit_Limit_Log                6075 non-null   float64
 12  Avg_Open_To_Buy_Sqrt            6075 non-null   float64
 13  Avg_Utilization_Ratio_Sqrt      6075 non-null   float64
 14  Gender_M                        6075 non-null   uint8  
 15  Education_Level_Doctorate       6075 non-null   uint8  
 16  Education_Level_Graduate        6075 non-null   uint8  
 17  Education_Level_High School     6075 non-null   uint8  
 18  Education_Level_Post-Graduate   6075 non-null   uint8  
 19  Education_Level_Uneducated      6075 non-null   uint8  
 20  Marital_Status_Married          6075 non-null   uint8  
 21  Marital_Status_Single           6075 non-null   uint8  
 22  Income_Category_$40K - $60K     6075 non-null   uint8  
 23  Income_Category_$60K - $80K     6075 non-null   uint8  
 24  Income_Category_$80K - $120K    6075 non-null   uint8  
 25  Income_Category_Less than $40K  6075 non-null   uint8  
 26  Income_Category_Less than 40K   6075 non-null   uint8  
 27  Card_Category_Gold              6075 non-null   uint8  
 28  Card_Category_Platinum          6075 non-null   uint8  
 29  Card_Category_Silver            6075 non-null   uint8  
dtypes: float64(5), int64(9), uint8(16)
memory usage: 806.8 KB

X_test.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2026 entries, 9760 to 413
Data columns (total 30 columns):
 #   Column                          Non-Null Count  Dtype  
---  ------                          --------------  -----  
 0   Customer_Age                    2026 non-null   int64  
 1   Dependent_count                 2026 non-null   int64  
 2   Months_on_book                  2026 non-null   int64  
 3   Total_Relationship_Count        2026 non-null   int64  
 4   Months_Inactive_12_mon          2026 non-null   int64  
 5   Contacts_Count_12_mon           2026 non-null   int64  
 6   Total_Revolving_Bal             2026 non-null   int64  
 7   Total_Amt_Chng_Q4_Q1            2026 non-null   float64
 8   Total_Trans_Amt                 2026 non-null   int64  
 9   Total_Trans_Ct                  2026 non-null   int64  
 10  Total_Ct_Chng_Q4_Q1             2026 non-null   float64
 11  Credit_Limit_Log                2026 non-null   float64
 12  Avg_Open_To_Buy_Sqrt            2026 non-null   float64
 13  Avg_Utilization_Ratio_Sqrt      2026 non-null   float64
 14  Gender_M                        2026 non-null   uint8  
 15  Education_Level_Doctorate       2026 non-null   uint8  
 16  Education_Level_Graduate        2026 non-null   uint8  
 17  Education_Level_High School     2026 non-null   uint8  
 18  Education_Level_Post-Graduate   2026 non-null   uint8  
 19  Education_Level_Uneducated      2026 non-null   uint8  
 20  Marital_Status_Married          2026 non-null   uint8  
 21  Marital_Status_Single           2026 non-null   uint8  
 22  Income_Category_$40K - $60K     2026 non-null   uint8  
 23  Income_Category_$60K - $80K     2026 non-null   uint8  
 24  Income_Category_$80K - $120K    2026 non-null   uint8  
 25  Income_Category_Less than $40K  2026 non-null   uint8  
 26  Income_Category_Less than 40K   2026 non-null   uint8  
 27  Card_Category_Gold              2026 non-null   uint8  
 28  Card_Category_Platinum          2026 non-null   uint8  
 29  Card_Category_Silver            2026 non-null   uint8  
dtypes: float64(5), int64(9), uint8(16)
memory usage: 269.1 KB

X_val.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2026 entries, 2894 to 6319
Data columns (total 30 columns):
 #   Column                          Non-Null Count  Dtype  
---  ------                          --------------  -----  
 0   Customer_Age                    2026 non-null   int64  
 1   Dependent_count                 2026 non-null   int64  
 2   Months_on_book                  2026 non-null   int64  
 3   Total_Relationship_Count        2026 non-null   int64  
 4   Months_Inactive_12_mon          2026 non-null   int64  
 5   Contacts_Count_12_mon           2026 non-null   int64  
 6   Total_Revolving_Bal             2026 non-null   int64  
 7   Total_Amt_Chng_Q4_Q1            2026 non-null   float64
 8   Total_Trans_Amt                 2026 non-null   int64  
 9   Total_Trans_Ct                  2026 non-null   int64  
 10  Total_Ct_Chng_Q4_Q1             2026 non-null   float64
 11  Credit_Limit_Log                2026 non-null   float64
 12  Avg_Open_To_Buy_Sqrt            2026 non-null   float64
 13  Avg_Utilization_Ratio_Sqrt      2026 non-null   float64
 14  Gender_M                        2026 non-null   uint8  
 15  Education_Level_Doctorate       2026 non-null   uint8  
 16  Education_Level_Graduate        2026 non-null   uint8  
 17  Education_Level_High School     2026 non-null   uint8  
 18  Education_Level_Post-Graduate   2026 non-null   uint8  
 19  Education_Level_Uneducated      2026 non-null   uint8  
 20  Marital_Status_Married          2026 non-null   uint8  
 21  Marital_Status_Single           2026 non-null   uint8  
 22  Income_Category_$40K - $60K     2026 non-null   uint8  
 23  Income_Category_$60K - $80K     2026 non-null   uint8  
 24  Income_Category_$80K - $120K    2026 non-null   uint8  
 25  Income_Category_Less than $40K  2026 non-null   uint8  
 26  Income_Category_Less than 40K   2026 non-null   uint8  
 27  Card_Category_Gold              2026 non-null   uint8  
 28  Card_Category_Platinum          2026 non-null   uint8  
 29  Card_Category_Silver            2026 non-null   uint8  
dtypes: float64(5), int64(9), uint8(16)
memory usage: 269.1 KB

def confusion_matrix_sklearn(model, predictors, target):
    """
    To plot the confusion_matrix with percentages

    model: classifier
    predictors: independent variables
    target: dependent variable
    """
    y_pred = model.predict(predictors)
    cm = confusion_matrix(target, y_pred)
    labels = np.asarray(
        [
            ["{0:0.0f}".format(item) + "\n{0:.2%}".format(item / cm.flatten().sum())]
            for item in cm.flatten()
        ]
    ).reshape(2, 2)

    plt.figure(figsize=(6, 4))
    sns.heatmap(cm, annot=labels, fmt="")
    plt.ylabel("True label")
    plt.xlabel("Predicted label")

# defining a function to compute different metrics to check performance of a classification model built using sklearn

def model_performance_classification_sklearn(model, predictors, target):
    """
    Function to compute different metrics to check classification model performance

    model: classifier
    predictors: independent variables
    target: dependent variable
    """

    # predicting using the independent variables
    pred = model.predict(predictors)

    acc = accuracy_score(target, pred)  # to compute Accuracy
    recall = recall_score(target, pred)  # to compute Recall
    precision = precision_score(target, pred)  # to compute Precision
    f1 = f1_score(target, pred)  # to compute F1-score

    # creating a dataframe of metrics
    df_perf = pd.DataFrame(
        {
            "Accuracy": acc,
            "Recall": recall,
            "Precision": precision,
            "F1": f1,
        },
        index=[0],
    )

    return df_perf

# Model 1: Logistic Regression

log_reg = LogisticRegression(random_state=1)
log_reg.fit(X_train, y_train)

LogisticRegression(random_state=1)

# I will evaluate the model performance by using KFold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=log_reg, X=X_train, y=y_train, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Performance on training set varies between approx. 0.33 to 0.53 recall.
• Next, I'll check the performance on validation data.

# Calculating different metrics on train set

log_reg_model_train_perf = model_performance_classification_sklearn(
    log_reg, X_train, y_train
)
print("Training performance:")
log_reg_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.871934	0.434426	0.652308	0.521525

# Calculating different metrics on validation set

log_reg_model_val_perf = model_performance_classification_sklearn(
    log_reg, X_val, y_val)
print("Validation performance:")
log_reg_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.880553	0.490798	0.677966	0.569395

# creating confusion matrix

confusion_matrix_sklearn(log_reg, X_val, y_val)

• Accuracy is high on both, and higher on validation set, so we aren't over-fitting the data
• Recall is quite low on both. I will try over-sampling to see if the model performance can be improved

# Over-Sampling with SMOTE

print("Before UpSampling, counts of label 'Yes': {}".format(sum(y_train == 1)))
print("Before UpSampling, counts of label 'No': {} \n".format(sum(y_train == 0)))

sm = SMOTE(
    sampling_strategy=1, k_neighbors=5, random_state=1
)  # Synthetic Minority Over Sampling Technique
X_train_over, y_train_over = sm.fit_resample(X_train, y_train)


print("After UpSampling, counts of label 'Yes': {}".format(sum(y_train_over == 1)))
print("After UpSampling, counts of label 'No': {} \n".format(sum(y_train_over == 0)))


print("After UpSampling, the shape of train_X: {}".format(X_train_over.shape))
print("After UpSampling, the shape of train_y: {} \n".format(y_train_over.shape))

Before UpSampling, counts of label 'Yes': 976
Before UpSampling, counts of label 'No': 5099 

After UpSampling, counts of label 'Yes': 5099
After UpSampling, counts of label 'No': 5099 

After UpSampling, the shape of train_X: (10198, 30)
After UpSampling, the shape of train_y: (10198,) 

# counts are now equal.
# Logistic Regression on Over-Sampled Data

log_reg_over = LogisticRegression(random_state=1)

# Training the basic logistic regression model with training set
log_reg_over.fit(X_train_over, y_train_over)

LogisticRegression(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_over = cross_val_score(
    estimator=log_reg_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_over)
plt.show()

• Recall on training set is now 84-88%, a vast improvement
• Now, I will see if this translates to success on the validation set

# Calculating different metrics on train set

log_reg_over_train_perf = model_performance_classification_sklearn(
    log_reg_over, X_train_over, y_train_over
)
print("Training performance:")
log_reg_over_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.840753	0.849971	0.834585	0.842208

# Calculating different metrics on validation set

log_reg_over_model_val_perf = model_performance_classification_sklearn(
    log_reg_over, X_val, y_val
)
print("validation performance:")
log_reg_over_model_val_perf

validation performance:

	Accuracy	Recall	Precision	F1
0	0.843534	0.843558	0.508318	0.634371

# creating confusion matrix

confusion_matrix_sklearn(log_reg_over, X_val, y_val)

• Recall has improved significantly, and the cost to the other scores is minimal.
• Model is not over-fitting
• Only 51 false negatives - this is good, but could be better

# Under-sampling with Random Under Sampler

rus = RandomUnderSampler(random_state=1)
X_train_un, y_train_un = rus.fit_resample(X_train, y_train)

print("Before Under Sampling, counts of label 'Yes': {}".format(sum(y_train == 1)))
print("Before Under Sampling, counts of label 'No': {} \n".format(sum(y_train == 0)))

print("After Under Sampling, counts of label 'Yes': {}".format(sum(y_train_un == 1)))
print("After Under Sampling, counts of label 'No': {} \n".format(sum(y_train_un == 0)))

print("After Under Sampling, the shape of train_X: {}".format(X_train_un.shape))
print("After Under Sampling, the shape of train_y: {} \n".format(y_train_un.shape))

Before Under Sampling, counts of label 'Yes': 976
Before Under Sampling, counts of label 'No': 5099 

After Under Sampling, counts of label 'Yes': 976
After Under Sampling, counts of label 'No': 976 

After Under Sampling, the shape of train_X: (1952, 30)
After Under Sampling, the shape of train_y: (1952,) 

# counts are now equal
# Logistic Regression on Under-Sampled Data

log_reg_under = LogisticRegression(random_state=1)
log_reg_under.fit(X_train_un, y_train_un)

LogisticRegression(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_under = cross_val_score(
    estimator=log_reg_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_under)
plt.show()

• Recall is now 79-85%, an improvement over the original, but not as good as oversampling.
• Checking performance on validation

# Calculating different metrics on train set

log_reg_under_train_perf = model_performance_classification_sklearn(
    log_reg_under, X_train_un, y_train_un
)
print("Training performance:")
log_reg_under_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.822746	0.83709	0.813745	0.825253

# Calculating different metrics on validation set

log_reg_under_model_val_perf = model_performance_classification_sklearn(
    log_reg_under, X_val, y_val
)
print("Validation performance:")
log_reg_under_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.816387	0.846626	0.461538	0.597403

# creating confusion matrix

confusion_matrix_sklearn(log_reg_under, X_val, y_val)

• Compared to oversampling, undersampling produces a better recall for the validation set, but worse for the training.
• We are still not over-fitting the data
• Only 50 false negatives - a marked improvement!

# Model 2: Decision Tree Classifier

dTree = DecisionTreeClassifier(criterion = 'gini', random_state=1)
dTree.fit(X_train, y_train)

DecisionTreeClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=dTree, X=X_train, y=y_train, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Decision tree gets a recall of 74-82% before over and under sampling
• That's markedly better than the logistic regression, but I suspect we are over-fitting. Getting scores next.

# Calculating different metrics on train set

dTree_model_train_perf = model_performance_classification_sklearn(
    dTree, X_train, y_train
)
print("Training performance:")
dTree_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

dTree_model_val_perf = model_performance_classification_sklearn(
    dTree, X_val, y_val
)
print("Validation performance:")
dTree_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.936328	0.803681	0.801223	0.80245

# creating confusion matrix

confusion_matrix_sklearn(dTree, X_val, y_val)

• As usual, the decision tree overfit the data
• Recall is better than the original logistic regression, but it doesn't really matter if we're overfitting
• 64 false negatives - not bad, but not great.

# Decision Tree on Over-Sampled Data
# Oversampling sets have already been created, applying decision tree to them

dTree_over = DecisionTreeClassifier(random_state=1)

# Training the basic logistic regression model with training set
dTree_over.fit(X_train_over, y_train_over)

DecisionTreeClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=dTree_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall with oversampling on a decision tree is 94-95% - a vast improvement
• Are we still overfitting the data?

# Calculating different metrics on train set

dTree_over_model_train_perf = model_performance_classification_sklearn(
    dTree_over, X_train_over, y_train_over
)
print("Training performance:")
dTree_over_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

dTree_over_model_val_perf = model_performance_classification_sklearn(
    dTree_over, X_val, y_val
)
print("validation performance:")
dTree_over_model_val_perf

validation performance:

	Accuracy	Recall	Precision	F1
0	0.929911	0.849693	0.748649	0.795977

# creating confusion matrix

confusion_matrix_sklearn(dTree_over, X_val, y_val)

• Recall is better than ever almost 85%
• 49 is the fewest false positives we've gotten
• All of this is moot, because we're still overfitting

# Decision tree on Under-Sampled data

dTree_under = DecisionTreeClassifier(random_state=1)
dTree_under.fit(X_train_un, y_train_un)

DecisionTreeClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=dTree_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall with decision tree under sampled is 88-91%
• I bet we're still overfitting

# Calculating different metrics on train set

dTree_under_model_train_perf = model_performance_classification_sklearn(
    dTree_under, X_train_un, y_train_un
)
print("Training performance:")
dTree_under_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

dTree_under_model_val_perf = model_performance_classification_sklearn(
    dTree_under, X_val, y_val
)
print("Validation performance:")
dTree_under_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.892892	0.898773	0.614256	0.729763

# creating confusion matrix

confusion_matrix_sklearn(dTree_under, X_val, y_val)

• Under-sampled decision tree gives the best recall so far at just under 90%
• 33 false negatives - that's the best so far
• Still moot, because we are over-fitting, but this might be a good candidate for tuning
• Since the decision tree is over-fitting, I will use the logistic regression as my base estimator for bagging

# Model 3: Bagging Classifier

bagging_lr=BaggingClassifier(base_estimator=LogisticRegression(random_state=1),random_state=1)
bagging_lr.fit(X_train,y_train)

BaggingClassifier(base_estimator=LogisticRegression(random_state=1),
                  random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=bagging_lr, X=X_train, y=y_train, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Bagging estimator gives a recall of 46-55%
• This improves on the logistic regression, but not the decision tree

# Calculating different metrics on train set

bagging_lr_model_train_perf = model_performance_classification_sklearn(
    bagging_lr, X_train, y_train
)
print("Training performance:")
bagging_lr_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.88642	0.492828	0.711538	0.582324

# Calculating different metrics on validation set

bagging_lr_model_val_perf = model_performance_classification_sklearn(
    bagging_lr, X_val, y_val
)
print("Validation performance:")
bagging_lr_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.896841	0.546012	0.74477	0.630088

# creating confusion matrix

confusion_matrix_sklearn(bagging_lr, X_val, y_val)

• The good news is that we aren't over-fitting
• The bad news is that the recall is only 54.6% - unacceptably low
• 148 false negatives
• I predict that this model will perform better with over- and under-sampling, just as occurred with the logistic regression

# Bagging Estimator on Over-Sampled Data

bagging_lr_over = BaggingClassifier(base_estimator=LogisticRegression(random_state=1),random_state=1)

# Training the bagging estimator with training set
bagging_lr_over.fit(X_train_over, y_train_over)

BaggingClassifier(base_estimator=LogisticRegression(random_state=1),
                  random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=bagging_lr_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall is 84-88%
• Vastly improved

# Calculating different metrics on train set

bagging_lr_over_model_train_perf = model_performance_classification_sklearn(
    bagging_lr_over, X_train_over, y_train_over
)
print("Training performance:")
bagging_lr_over_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.843008	0.858207	0.832889	0.845359

# Calculating different metrics on validation set

bagging_lr_over_model_val_perf = model_performance_classification_sklearn(
    bagging_lr_over, X_val, y_val
)
print("Validation performance:")
bagging_lr_over_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.837611	0.846626	0.497297	0.626561

# creating confusion matrix

confusion_matrix_sklearn(bagging_lr_over, X_val, y_val)

• Recall of 84.6% is incredible
• Only 50 false negatives - great!
• Still not over-fitting.

# Bagging Estimator on Under-Sampled Data

bagging_lr_under = BaggingClassifier(base_estimator=LogisticRegression(random_state=1),random_state=1)

# Training the bagging estimator with training set
bagging_lr_under.fit(X_train_un, y_train_un)

BaggingClassifier(base_estimator=LogisticRegression(random_state=1),
                  random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=bagging_lr_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall is 80-86%
• Over-sampling performed better in this case

# Calculating different metrics on train set

bagging_lr_under_model_train_perf = model_performance_classification_sklearn(
    bagging_lr_under, X_train_un, y_train_un
)
print("Training performance:")
bagging_lr_under_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.824283	0.838115	0.815553	0.82668

# Calculating different metrics on validation set

bagging_lr_under_model_val_perf = model_performance_classification_sklearn(
    bagging_lr_under, X_val, y_val
)
print("Validation performance:")
bagging_lr_under_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.815893	0.837423	0.460371	0.594124

# creating confusion matrix

confusion_matrix_sklearn(bagging_lr_under, X_val, y_val)

• We still aren't over-fitting
• All scoring metrics are lower than under-sampled bagging estimator
• Recall is now 83.7%, worse than under-sampled bagging estimator
• Overall, the best bagging estimator is the under-sampled one, a prime candidate for tuning.

# Model 4: Random Forest Estimator

rf_estimator=RandomForestClassifier(random_state=1)
rf_estimator.fit(X_train,y_train)

RandomForestClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=rf_estimator, X=X_train, y=y_train, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Random Forest gives a recall of 71-77%, a good start
• I'd be willing to bet this is over-fitting

# Calculating different metrics on train set

rf_estimator_model_train_perf = model_performance_classification_sklearn(
    rf_estimator, X_train, y_train
)
print("Training performance:")
rf_estimator_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

rf_estimator_model_val_perf = model_performance_classification_sklearn(
    rf_estimator, X_val, y_val
)
print("Validation performance:")
rf_estimator_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.956071	0.794479	0.921708	0.853377

# creating confusion matrix

confusion_matrix_sklearn(rf_estimator, X_val, y_val)

• Over-fitting the data
• Recall is low at 79.4%
• 67 false negatives - too high
• I suspect that we will still over-fit with under- and over-sampled data, but I will try

# Random Forest Estimator on Over-Sampled Data

rf_estimator_over=RandomForestClassifier(random_state=1)
rf_estimator_over.fit(X_train_over,y_train_over)

RandomForestClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=rf_estimator_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Over-sampled recall is 97-99% - best yet
• But is it over-fit?

# Calculating different metrics on train set

rf_estimator_over_model_train_perf = model_performance_classification_sklearn(
    rf_estimator_over, X_train_over, y_train_over
)
print("Training performance:")
rf_estimator_over_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

rf_estimator_over_model_val_perf = model_performance_classification_sklearn(
    rf_estimator_over, X_val, y_val
)
print("Validation performance:")
rf_estimator_over_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.956565	0.861963	0.867284	0.864615

# creating confusion matrix

confusion_matrix_sklearn(rf_estimator_over, X_val, y_val)

• Recall is good at 86%
• False negatives are low at 45
• However, we are still over-fit.

# Random Forest Estimator on Under-Sampled Data

rf_estimator_under=RandomForestClassifier(random_state=1)
rf_estimator_under.fit(X_train_un,y_train_un)

RandomForestClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=rf_estimator_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall is 91-97%, which is great
• It's probably overfitting

# Calculating different metrics on train set

rf_estimator_under_model_train_perf = model_performance_classification_sklearn(
    rf_estimator_under, X_train_un, y_train_un
)
print("Training performance:")
rf_estimator_under_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

rf_estimator_under_model_val_perf = model_performance_classification_sklearn(
    rf_estimator_under, X_val, y_val
)
print("Validation performance:")
rf_estimator_under_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.928924	0.920245	0.717703	0.806452

# creating confusion matrix

confusion_matrix_sklearn(rf_estimator_under, X_val, y_val)

• 92% recall is amazing!
• Only 26 false negatives
• Unfortunately, we are over-fitting
• This is an excellent candidate for tuning

# Model 5: Gradient Boosting Classifier

gbc = GradientBoostingClassifier(random_state=1)
gbc.fit(X_train,y_train)

GradientBoostingClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=gbc, X=X_train, y=y_train, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall is 79-82%, which is a great place to start
• Are we overfitting?

# Calculating different metrics on train set

gbc_model_train_perf = model_performance_classification_sklearn(
    gbc, X_train, y_train
)
print("Training performance:")
gbc_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.97284	0.875	0.952062	0.911906

# Calculating different metrics on validation set
gbc_model_val_perf = model_performance_classification_sklearn(
    gbc, X_val, y_val
)
print("Validation performance:")
gbc_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.967423	0.855828	0.936242	0.894231

# creating confusion matrix

confusion_matrix_sklearn(gbc, X_val, y_val)

• 85.6% recall is a great place to start
• So is 47 false negatives
• The high accuracy on the training data would usually indicate over-fitting, but, in this case, the validation accuracy is almost as high, which tells me we just have a good model

# Gradient Boosting Classifier on Over-Sampled Data

gbc_over=GradientBoostingClassifier(random_state=1)
gbc_over.fit(X_train_over,y_train_over)

GradientBoostingClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=gbc_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Training recall is 97-98% - amazing!

# Calculating different metrics on train set

gbc_over_model_train_perf = model_performance_classification_sklearn(
    gbc_over, X_train_over, y_train_over
)
print("Training performance:")
gbc_over_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.979212	0.982546	0.976037	0.979281

# Calculating different metrics on validation set

gbc_over_model_val_perf = model_performance_classification_sklearn(
    gbc_over, X_val, y_val
)
print("Validation performance:")
gbc_over_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.9615	0.898773	0.866864	0.88253

# creating confusion matrix

confusion_matrix_sklearn(gbc_over, X_val, y_val)

• These results are breathtaking
• Recall is just under 90%
• Only 33 false negatives
• Not over-fitting

# Gradient Boosting Classifier on Under-Sampled Data

gbc_under=GradientBoostingClassifier(random_state=1)
gbc_under.fit(X_train_un,y_train_un)

GradientBoostingClassifier(random_state=1)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=gbc_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Training recall is 92-96% - not as good as undersampled GRC

# Calculating different metrics on train set

gbc_under_model_train_perf = model_performance_classification_sklearn(
    gbc_under, X_train_un, y_train_un
)
print("Training performance:")
gbc_under_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	0.973873	0.979508	0.968592	0.974019

# Calculating different metrics on validation set

gbc_under_model_val_perf = model_performance_classification_sklearn(
    gbc_under, X_val, y_val
)
print("Validation performance:")
gbc_under_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.936821	0.957055	0.732394	0.829787

# creating confusion matrix

confusion_matrix_sklearn(gbc_under, X_val, y_val)

• Only 14 false negatives
• 95.7% recall
• Not overfitting
• So far, this is the best candidate for tuning

# Model 6: Extreme Gradient Boosting

xgb = XGBClassifier(random_state=1,eval_metric='logloss')
xgb.fit(X_train,y_train)

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, eval_metric='logloss',
              gamma=0, gpu_id=-1, importance_type='gain',
              interaction_constraints='', learning_rate=0.300000012,
              max_delta_step=0, max_depth=6, min_child_weight=1, missing=nan,
              monotone_constraints='()', n_estimators=100, n_jobs=4,
              num_parallel_tree=1, random_state=1, reg_alpha=0, reg_lambda=1,
              scale_pos_weight=1, subsample=1, tree_method='exact',
              validate_parameters=1, verbosity=None)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=xgb, X=X_train, y=y_train, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Very strange box plot - seems to show a recall of around 87%

# Calculating different metrics on train set

xgb_model_train_perf = model_performance_classification_sklearn(
    xgb, X_train, y_train
)
print("Training performance:")
xgb_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

xgb_model_val_perf = model_performance_classification_sklearn(
    xgb, X_val, y_val
)
print("Validation performance:")
xgb_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.968411	0.886503	0.914557	0.900312

# creating confusion matrix

confusion_matrix_sklearn(xgb, X_val, y_val)

• Clearly over-fitting
• Nice start with 37 false negatives
• Recall on validation is 88.7%

# Extreme Gradient Boosting on Over-Sampled Data

xgb_over = XGBClassifier(random_state=1,eval_metric='logloss')
xgb_over.fit(X_train_over,y_train_over)

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, eval_metric='logloss',
              gamma=0, gpu_id=-1, importance_type='gain',
              interaction_constraints='', learning_rate=0.300000012,
              max_delta_step=0, max_depth=6, min_child_weight=1, missing=nan,
              monotone_constraints='()', n_estimators=100, n_jobs=4,
              num_parallel_tree=1, random_state=1, reg_alpha=0, reg_lambda=1,
              scale_pos_weight=1, subsample=1, tree_method='exact',
              validate_parameters=1, verbosity=None)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=xgb_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Training recall is in the 98-99% range - great, but is it overfit?

# Calculating different metrics on train set

xgb_over_model_train_perf = model_performance_classification_sklearn(
    xgb_over, X_train_over, y_train_over
)
print("Training performance:")
xgb_over_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

xgb_over_model_val_perf = model_performance_classification_sklearn(
    xgb_over, X_val, y_val
)
print("Validation performance:")
xgb_over_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.968904	0.91411	0.894895	0.904401

# creating confusion matrix

confusion_matrix_sklearn(xgb_over, X_val, y_val)

• Still over-fitting
• Recall improved to 91.4%
• Only 28 false negatives

# Extreme Gradient Boosting with Under-Sampled Data

xgb_under = XGBClassifier(random_state=1,eval_metric='logloss')
xgb_under.fit(X_train_un,y_train_un)

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, eval_metric='logloss',
              gamma=0, gpu_id=-1, importance_type='gain',
              interaction_constraints='', learning_rate=0.300000012,
              max_delta_step=0, max_depth=6, min_child_weight=1, missing=nan,
              monotone_constraints='()', n_estimators=100, n_jobs=4,
              num_parallel_tree=1, random_state=1, reg_alpha=0, reg_lambda=1,
              scale_pos_weight=1, subsample=1, tree_method='exact',
              validate_parameters=1, verbosity=None)

# K-Fold

scoring = "recall"
kfold = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=1
)  # Setting number of splits equal to 5
cv_result_bfr = cross_val_score(
    estimator=xgb_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold
)
# Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()

• Recall on training is 94-97%
• I would assume we are still over-fitting

# Calculating different metrics on train set

xgb_under_model_train_perf = model_performance_classification_sklearn(
    xgb_under, X_train_un, y_train_un
)
print("Training performance:")
xgb_under_model_train_perf

Training performance:

	Accuracy	Recall	Precision	F1
0	1.0	1.0	1.0	1.0

# Calculating different metrics on validation set

xgb_under_model_val_perf = model_performance_classification_sklearn(
    xgb_under, X_val, y_val
)
print("Validation performance:")
xgb_under_model_val_perf

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.940276	0.947853	0.748184	0.836265

# creating confusion matrix

confusion_matrix_sklearn(xgb_under, X_val, y_val)

Observations

• Clearly still overfitting
• Only 17 false negatives
• 94.8% recall on the validation set
• This is a top candidate for tuning

# Validation performance comparison

models_val_comp_df = pd.concat(
    [
        log_reg_model_val_perf.T, 
        log_reg_over_model_val_perf.T, 
        log_reg_under_model_val_perf.T, 
        dTree_model_val_perf.T, 
        dTree_over_model_val_perf.T, 
        dTree_under_model_val_perf.T,
        bagging_lr_model_val_perf.T, 
        bagging_lr_over_model_val_perf.T, 
        bagging_lr_under_model_val_perf.T, 
        rf_estimator_model_val_perf.T, 
        rf_estimator_over_model_val_perf.T, 
        rf_estimator_under_model_val_perf.T, 
        gbc_model_val_perf.T, 
        gbc_over_model_val_perf.T, 
        gbc_under_model_val_perf.T, 
        xgb_model_val_perf.T, 
        xgb_over_model_val_perf.T, 
        xgb_under_model_val_perf.T
    ],
    axis=1,
)
models_val_comp_df.columns = [
    "Logistic Regression",
    "Logistic Regression with oversampled data",
    "Logistic Regression with undersampled data",
    "Decision Tree",
    "Decision Tree with oversampled data",
    "Decision Tree with undersampled data",
    "Bagging Estimator",
    "Bagging Estimator with oversampled data",
    "Bagging Estimator with undersampled data",
    "Random Forest Estimator",
    "Random Forest Estimator with oversampled data",
    "Random Forest Estimator with undersampled data",
    "Gradient Boosting Classifier",
    "Gradient Boosting Classifier with oversampled data",
    "Gradient Boosting Classifier with undersampled data",
    "Extreme Gradient Boosting",
    "Extreme Gradient Boosting with oversampled data",
    "Extreme Gradient Boosting with undersampled data",
]
print("Validation performance comparison:")
models_val_comp_df

Validation performance comparison:

	Logistic Regression	Logistic Regression with oversampled data	Logistic Regression with undersampled data	Decision Tree	Decision Tree with oversampled data	Decision Tree with undersampled data	Bagging Estimator	Bagging Estimator with oversampled data	Bagging Estimator with undersampled data	Random Forest Estimator	Random Forest Estimator with oversampled data	Random Forest Estimator with undersampled data	Gradient Boosting Classifier	Gradient Boosting Classifier with oversampled data	Gradient Boosting Classifier with undersampled data	Extreme Gradient Boosting	Extreme Gradient Boosting with oversampled data	Extreme Gradient Boosting with undersampled data
Accuracy	0.880553	0.843534	0.816387	0.936328	0.929911	0.892892	0.896841	0.837611	0.815893	0.956071	0.956565	0.928924	0.967423	0.961500	0.936821	0.968411	0.968904	0.940276
Recall	0.490798	0.843558	0.846626	0.803681	0.849693	0.898773	0.546012	0.846626	0.837423	0.794479	0.861963	0.920245	0.855828	0.898773	0.957055	0.886503	0.914110	0.947853
Precision	0.677966	0.508318	0.461538	0.801223	0.748649	0.614256	0.744770	0.497297	0.460371	0.921708	0.867284	0.717703	0.936242	0.866864	0.732394	0.914557	0.894895	0.748184
F1	0.569395	0.634371	0.597403	0.802450	0.795977	0.729763	0.630088	0.626561	0.594124	0.853377	0.864615	0.806452	0.894231	0.882530	0.829787	0.900312	0.904401	0.836265

Observations

• Overall, the under-sampled data performed the best
• Recall came in over 90% in the following three models:
  • Random Forest Estimator Under-Sampled: 92.0%
  • Gradient Boosting Classifier Under-Sampled: 95.7%
  • Extreme Gradient Boosting Under-Sampled: 94.8%
• Of these, XGB and Random Forest are both over-fit. However, I would like to see if fine-tuning can eliminate that.

Regularization with RandomizedSearchCV

# I will start by tuning the Extreme Gradient Boosting Classifier

import sklearn.metrics as metrics
from sklearn.model_selection import RandomizedSearchCV

# defining model
model1 = XGBClassifier(random_state=1,eval_metric='logloss')

# Parameter grid to pass in RandomizedSearchCV
param_grid={'n_estimators':np.arange(50,150,50),
            'scale_pos_weight':[2,5,10],
            'learning_rate':[0.01,0.1,0.2,0.05],
            'gamma':[0,1,3,5],
            'subsample':[0.8,0.9,1],
            'max_depth':np.arange(1,5,1),
            'reg_lambda':[5,10]}

# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)

#Calling RandomizedSearchCV
xgb_under_tuned = RandomizedSearchCV(estimator=model1, param_distributions=param_grid, n_iter=50, scoring=scorer, cv=5, random_state=1, n_jobs = -1)

#Fitting parameters in RandomizedSearchCV
xgb_under_tuned.fit(X_train_un,y_train_un)

print("Best parameters are {} with CV score={}:" .format(xgb_under_tuned.best_params_,xgb_under_tuned.best_score_))

Best parameters are {'subsample': 0.9, 'scale_pos_weight': 10, 'reg_lambda': 5, 'n_estimators': 50, 'max_depth': 1, 'learning_rate': 0.01, 'gamma': 1} with CV score=1.0:

# Calculating different metrics on train set
xgboost_random_train_under = model_performance_classification_sklearn(
    xgb_under_tuned, X_train_un, y_train_un
)
print("Training performance:")
xgboost_random_train_under

Training performance:

	Accuracy	Recall	Precision	F1
0	0.5	1.0	0.5	0.666667

# Calculating different metrics on validation set
xgboost_random_val_under = model_performance_classification_sklearn(
    xgb_under_tuned, X_val, y_val
)
print("Validation performance:")
xgboost_random_val_under

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.160908	1.0	0.160908	0.277211

# creating confusion matrix

confusion_matrix_sklearn(xgb_under_tuned, X_val, y_val)

• Great results on recall, but only recall
• This model over-fits like crazy, and Accuracy, Precision and F1 are nowhere near acceptable
• 0 False negatives. But it's just too good to be true.

# Next, I will tune the Random Forest Classifier

# defining model
model2 = RandomForestClassifier(random_state=1)

# Grid of parameters to choose from
# add from article
param_grid2 = {"n_estimators": [150,200,250],
    "min_samples_leaf": np.arange(5, 10),
    "max_features": np.arange(0.2, 0.7, 0.1),
    "max_samples": np.arange(0.3, 0.7, 0.1),
             }

# Type of scoring used to compare parameter combinations
scorer2 = metrics.make_scorer(metrics.recall_score)

#Calling RandomizedSearchCV
rf_estimator_under_tuned = RandomizedSearchCV(estimator=model2, param_distributions=param_grid2, n_iter=50, scoring=scorer, cv=5, random_state=1, n_jobs = -1)

#Fitting parameters in RandomizedSearchCV
rf_estimator_under_tuned.fit(X_train_un,y_train_un)

print("Best parameters are {} with CV score={}:" .format(rf_estimator_under_tuned.best_params_,rf_estimator_under_tuned.best_score_))
rf_estimator_tuned = RandomForestClassifier(random_state=1)

Best parameters are {'n_estimators': 200, 'min_samples_leaf': 6, 'max_samples': 0.6000000000000001, 'max_features': 0.5000000000000001} with CV score=0.9436630036630037:

# Calculating different metrics on train set
rf_estimator_random_train_under = model_performance_classification_sklearn(
    rf_estimator_under_tuned, X_train_un, y_train_un
)
print("Training performance:")
rf_estimator_random_train_under

Training performance:

	Accuracy	Recall	Precision	F1
0	0.961578	0.976434	0.948259	0.96214

# Calculating different metrics on validation set
rf_estimator_random_val_under = model_performance_classification_sklearn(
    rf_estimator_under_tuned, X_val, y_val
)
print("Validation performance:")
rf_estimator_random_val_under

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.915597	0.95092	0.666667	0.783818

# creating confusion matrix

confusion_matrix_sklearn(rf_estimator_under_tuned, X_val, y_val)

• Best performance yet. Only 16 false positives
• Recall is 95.1% on the validation set
• Not over-fitting.
• The only issue is precision - it is still under 70%

# Finally, I will tune the Gradient Boosting Classifier

# defining model
model3 = GradientBoostingClassifier(random_state=1)

# Grid of parameters to choose from
# add from article
param_grid3 = {
    "n_estimators": [100,150,200,250],
    "subsample":[0.8,0.9,1],
    "max_features":[0.7,0.8,0.9,1]
}

# Type of scoring used to compare parameter combinations
scorer3 = metrics.make_scorer(metrics.recall_score)

#Calling RandomizedSearchCV
gbc_under_tuned = RandomizedSearchCV(estimator=model3, param_distributions=param_grid3, n_iter=50, scoring=scorer3, cv=5, random_state=1, n_jobs = -1)

#Fitting parameters in RandomizedSearchCV
gbc_under_tuned.fit(X_train_un,y_train_un)

print("Best parameters are {} with CV score={}:" .format(gbc_under_tuned.best_params_,gbc_under_tuned.best_score_))

Best parameters are {'subsample': 0.9, 'n_estimators': 250, 'max_features': 0.8} with CV score=0.959010989010989:

# Calculating different metrics on train set
gbc_random_train_under = model_performance_classification_sklearn(
    gbc_under_tuned, X_train_un, y_train_un
)
print("Training performance:")
gbc_random_train_under

Training performance:

	Accuracy	Recall	Precision	F1
0	0.997439	1.0	0.994903	0.997445

# Calculating different metrics on validation set
gbc_random_val_under = model_performance_classification_sklearn(
    gbc_under_tuned, X_val, y_val
)
print("Validation performance:")
gbc_random_val_under

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.947187	0.972393	0.763855	0.855601

# creating confusion matrix

confusion_matrix_sklearn(gbc_under_tuned, X_val, y_val)

• Only 9 false negatives, and a recall on validation of 97.2%
• Not over-fitting
• All metrics are within acceptable limits, we have a winner!

# building model with best parameters
Final_Model = GradientBoostingClassifier(
    subsample = 0.9,
    n_estimators = 250,
    max_features = 0.8
)
# Fit the model on training data
Final_Model.fit(X_train_un, y_train_un)

GradientBoostingClassifier(max_features=0.8, n_estimators=250, subsample=0.9)

Final_Model_Train = model_performance_classification_sklearn( Final_Model, X_train_un, y_train_un ) print("Training performance:") Final_Model_Train

Final_Model_Train = model_performance_classification_sklearn(
    Final_Model, X_train_un, y_train_un
)
print("Train performance:")
Final_Model_Train

Train performance:

	Accuracy	Recall	Precision	F1
0	0.997439	1.0	0.994903	0.997445

Final_Model_Val = model_performance_classification_sklearn(
    Final_Model, X_val, y_val
)
print("Validation performance:")
Final_Model_Val

Validation performance:

	Accuracy	Recall	Precision	F1
0	0.949161	0.969325	0.772616	0.859864

# Finally, we can measure performance on the test data

Final_Model_Test = model_performance_classification_sklearn(
    Final_Model, X_test, y_test
)
print("Test performance:")
Final_Model_Test

Test performance:

	Accuracy	Recall	Precision	F1
0	0.945212	0.966154	0.758454	0.849797

• Unsurprisingly, similar results on test data as we found on validation

Importances

feature_names = X_test.columns
importances = Final_Model.feature_importances_
indices = np.argsort(importances)

plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()

• "Total_Trans_Ct" is the most important variable
• This is followed, respectively, by "Total_Trans_Amt", "Total_Revolving_Bal", "Total_Ct_Chng_Q4_Q1", "Total_Relationship_Ct", and "Total_Amt_Chng_Q4_Q1"

Pipelines

• Now we have our final model - I'll use pipelines to put it into production
• Using Column transformer, I will create 2 different pipelines, one for numerical columns and one for categorical columns
• For categorical columns, I will do one hot encoding and missing value imputation as pre-processing
• For numerical columns, there is no need for missing value imputation since no numerical columns were missing data

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

# creating a transformer for categorical variables, which will first apply simple imputer and 
# then do one hot encoding for categorical variables
categorical_features = ["Gender", "Education_Level", "Marital_Status", 
                        "Income_Category", "Card_Category"]

categorical_transformer = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore")),
    ]
)
preprocessor = ColumnTransformer(
    transformers=[
        ("cat", categorical_transformer, categorical_features),
    ],
    remainder="passthrough",
)

X = df.drop(columns="Attrition_Flag")
Y = df["Attrition_Flag"]

# We already know the best model, so we don't need a validation set
X_train, X_test, y_train, y_test = train_test_split(
    X, Y, test_size=0.30, random_state=1, stratify=Y
)
print(X_train.shape, X_test.shape)

(7088, 19) (3039, 19)

# Under-sampling with Random Under Sampler

rus = RandomUnderSampler(random_state=1)
X_train_un, y_train_un = rus.fit_resample(X_train, y_train)

# Creating new pipeline with best parameters
pipe = Pipeline(
    steps=[
        ("pre", preprocessor),
        (
            "GBC",
            GradientBoostingClassifier(
    subsample = 0.9,
    n_estimators = 250,
    max_features = 0.8
        ),
        )
    ]
)
# Fit the model on training data
pipe.fit(X_train_un, y_train_un)

Pipeline(steps=[('pre',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('cat',
                                                  Pipeline(steps=[('imputer',
                                                                   SimpleImputer(strategy='most_frequent')),
                                                                  ('onehot',
                                                                   OneHotEncoder(handle_unknown='ignore'))]),
                                                  ['Gender', 'Education_Level',
                                                   'Marital_Status',
                                                   'Income_Category',
                                                   'Card_Category'])])),
                ('GBC',
                 GradientBoostingClassifier(max_features=0.8, n_estimators=250,
                                            subsample=0.9))])

Pipe_Test = model_performance_classification_sklearn(
    pipe, X_test, y_test
)
print("Test performance:")
Pipe_Test

Test performance:

	Accuracy	Recall	Precision	F1
0	0.952287	0.969262	0.784411	0.867094

# creating confusion matrix

confusion_matrix_sklearn(pipe, X_test, y_test)

• Final results are magnificent
• 19 false negatives
• On test data:
  • 96.1% Recall score
  • 95.3% Accuracy score
  • 79.0% Precision score
  • 86.7% F1 score

Business Recommendations

• "Total_Trans_Ct" is the most important variable. "Total_Trans_Amt" is 2nd most important, and goes hand in hand with "Total_Trans_Ct". Therefore, I would recommend that the bank flag customers who have a low number of transactions, and a low amount of transactions. These customers should be contacted to gage their satisfaction with their credit card, in an attempt to retain them as customers.
• "Total_Revolving_Bal" is the next most important variable, followed by "Total_Ct_Chng_Q4_Q1", and "Total_Amt_Chng_Q4_Q1". It would be helpful to identify customers with low values in these categories, and again reach out to retain them as customers.
• "Total_Relationship_Ct" is highly important. This makes it obvious to me that customers with more banking products are less likely to drop their credit card. Therefore, it would be helpful to identify customers that have only 1 or 2 accounts with the bank, and attempt to establish more relationships with them.